SEP 1 S 2005 



03 



IN THE MATTER OF 
KOREAN PATENT APPLICATION 
UNDER SERIAL NO. 3 1 84/1 999 



I, THE UNDERSIGNED, HEREBY DECLARE : 

THAT I AM CONVERSANT WITH BOTH THE KOREAN AND THE ENGLISH 
LANGUAGES : AND 

THAT I AM A COMPETENT TRANSLATOR OF THE APPLICATION PAPERS THE 
PARTICULARS OF WHICH ARE SET FORTH BELOW : 



KOREAN PATENT APPLICATION UNDER 
SERIAL NO.: 3184/1999 



FILED ON : 



February 1, 1999 



IN THE NAME OF 



LG ELECTRONICS INC. 



FOR 



DATA STRUCTURE OF MULTI-LEVEL 
IMAGE AND METHOD FOR IMAGE 
CHECKING USING THIS STRUCTURE 



IN WITNESS WHEREOF, I SET MY HAND HERETO 
THIS 9TH DAY OF SEPTEMBER, 2005 

BY 




Ye Jin KIM 



[Translation] 



PATENT APPLICATION 



To : Director General 
The Patent Office 



Date of Application : February 1, 1999 

Title of the Invention : DATA STRUCTURE OF MULTI-LEVEL IMAGE AND 
METHOD FOR IMAGE CHECKING USING THIS STRUCTURE 



Applicant : LG ELECTRONICS INC. 

Code No. : 1-1998-000275-8 

Attorney : Name : CHOI, Yeong Bok 

Code No. : 9-1998-000571-2 



Inventor : Name : KIM, Hyeon Jun 

Resident Reg. No.: 640904-1117118 
Zip code : 463-030 

Address : Hanshin-Life 109-302. Bundancr-TV™ 

' w ' O? — ' '"'"""fc v_*« 5 

Seongnam, Gyeonggi-Do, Korea 
Nationality : Republic of Korea 

Name : JUN, Sung Bae 

Resident Reg. No.: 711010-1057913 

Zip code : 153-034 

Address : 804, Siheung 4-Dong, Geumcheon-Gu, Seoul, Korea 
Nationality : Republic of Korea 

Name : LEE, Jin Soo 

Resident Reg. No.: 710502-1080034 

Zip code : 138-122 

Address : Samik Apt. 101-804, 573, Macheon 2-Dong, Songpa-Gu, 

Seoul, Korea 
Nationality : Republic of Korea 



Request for Examination : Filed 



This application is hereby filed pursuant to Article 42 of the Patent Law and 
request for examination is filed pursuant to Article 60 of the Patent Law. 



/S/ Attorney : CHOI, Yeong Bok 



[Fee] 
Basic filing fee 
Additional filing fee 
Fee for claiming a priority - 
Fee for filing request for examinationl 
Total - 



20 Pages 29,000 WON 
3 Pages 3,000 WON 
0 Case 0 WON 

13 Claims 525,000 WON 
557,000 WON 



[Attached document] 
1. Abstract, Specification (Drawing) - lcopy 



Ends. 




[Translation] 

ABSTRACT OF THE DISCLOSURE 



[Abstract] 

The present invention relates to an image search method capable of expressing 
one color feature related to a spatial color feature of a still image based on a multilevel 
image grid and searching target images using the expressed multilevel image grid. In the 
conventional art, since one color feature is expressed based on one image grid level. In 
tins case, the importance of each feature is expressed but an element-based importance of 
each feature is not adopted. In addition, since the element-based average importance is 
■various based on the characteristic of the reference image or target images, it is 
impossible to effectively response for a user's contents-based image search. In the 
present invention, one feature has a multilevel image grid, and each level is formed of 
cells of hierarchical grids of different levels for thereby obtaining a data structure in 
which each cell corresponding to the grid is expressed based on a reliability on a region 
representative color and a similarity of the region representative color, so that it is 
possible to fast and accurately search images with respect to a user's content-based query 
based on a cell matching of the same level as two image grids and different levels or a 
color local matching of the grid match. 
[Representative Drawing] 
Figure 1 



1 



[SPECIFICATION] 



[Title of the Invention] 

Data structure of multi-level image and method for image checking using this structure 
[Brief description of the Drawings] 

Fig. 1 shows an embodiment of a multi-level image data structure according to the 
present invention, which shows a 3-leveI image data structure; 

Fig. 2 shows a match between 3-leveI image grids in an image detecting method 
using a multi-level image data structure according to the present invention; 

Fig. 3 shows a match between the same levels in the match between the 3-Ieve] 
image gnus shown in the embodiment ol Fig. 2; 

Fig. 4 shows the embodiment of Fig. 2, (a) showing two identical image grid 
levels, and (b) showing a match between two image grids; and 

Fig. 5 shows a match between two different image grid levels in the embodiment 
of Fig. 2. 

[Detailed description of the invention] 

[Object of the invention] 

[Field of the invention and background art] 

The present invention is directed to an image search method using a multilevel 
image grid by which one color feature related to a spatial color property of a still image is 
expressed. 

In a conventional image search method, features such as color, shape, texture, and 
the like are expressed in an image grid data structure of one level, and similarity of the 
features are searched. 
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At this time, the importance of each feature is different according to the 
characteristics of an image to be searched. In addition, even with respect to only one 
feature, the importance thereof is different according to parts of cells existing in its grid. 

For instance, in the image search method using a color histogram, which is 
formed in a n-dimensional structure, a weight value can be determined as a different 
value for each element forming the n-dimensional structure. 

Thus, in the conventional method, the importance between features has been 
expressed based on the image grid expressed by one level for one image. However. The 
importance for each element within the independent feature has not been considered. 
Also, another method for previously calculating the average importance for each element 
fur determination has been used. 

The average importance for each element varies by a reference image or a target 
image. Accordingly, the determining of the average value for elements is not useful in 
image search. 

In addition, since the image grid data structure is formed of only one level, data 
for the image grid is not accurately searched. As a result, it is impossible to carry out an 
optimal search for a user's query. 

[Technical object of the present invention] 

Therefore, in order to solve the above problem, an object of the present invention 
is to provide a data structure in which one feature has a multilevel image grid, each of 
which levels is allowed to be formed of cells of a hierarchical structure of different levels, 
and each cell is expressed by a region representative color and a reliability expressing a 
similarity with respect to the region representative color value. 

Another object of the present invention is to provide an image search method 
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capable of effectively corresponding to a user's query by cells of the same level of two 
image grids and different levels or a grid match and a color region match to perform the 
searching of a color similarity with respect to different multilevel image grids 
corresponding to reference destination image. 

[Construction of the present invention] 

To achieve the above objects according to the present invention, there is provided 
a multilevel image data structure in which a spatial color feature of one image is 
expressed in a hierarchical image grid structure having at least two or more different 
levels. 

The hierarchical image grid of each level is characterized by hierarchically- 
dividing resolution of cells in proportion to height and width of the corresponding image 
and thus hierarchically improving the resolution of the cells of each level. 

For square image (quad tree structure), the hierarchical level of each image grid is 
hierarchically divided by uniformly dividing an aspect ratio (height: width), whereas for 
non-square image (non-quad tree structure), one side of the aspect ratio is uniformly 
divided, and the other side thereof is hierarchically divided by the unit of the one side. 

An image search method using such multilevel image data structure is 
characterized by which region representative colors with respect to regions between 
different images divided into multilevel are matched to reliability indicating accuracy of 
the representative color value, and thus color similarity can be searched according to a 
user content-based query. 

The color similarity between two images having different hierarchical grids may 
be searched according to difference of similarities for representative color values of two 
color points having a spatial color feature by matching each cell included in two different 
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image grids, searched by matching two image grids, performing a multi-cross in 
accordance with a spatial color feature between the two images, and searching color 
similarities, and searched by matching each region representative color value for thereby 
searching the same regions. 

Such image search method using the multilevel image data will now be explained 
with reference to Figs. 1 to 5. 

First, one image data structure is expressed in a multilevel image grid structure in 
which the spatial color feature is divided into hierarchical grids of different levels. 

Each image grid is a hierarchical structure of different levels, and the resolution of 
each level is hierarchically divided. The cell of each grid is assigned with two values 
which are a region represcinmive coior (RRC) and a reliability score (S) related to 
accuracy of the region representative color. 

For a quad tree structure, an aspect ratio of the multilevel image grid is uniformly 
divided, and for non-quad tree structure, one side is uniformly divided in accordance with 
an aspect ratio of a width and height of an image, and the other side is uniformly divided 
by the unit of the one side. 

That is, a regular square structure having the same length of horizontal and 
vertical sides is divided by the same unit with respect to the aspect ratio, and for a 
rectangular structure having different lengths of horizontal and vertical sides, one side 
(e.g., a lengthy side) is uniformly divided, and the other side (e.g., a shorter side) is 
divided by the dividing unit of the one side. 

Fig. 1 illustrates an embodiment of a multilevel image grid data structure 
according to the present invention, which shows a 3-level grid data structure. The 3-Ievel 
image grid is divided into image grids of top (first) level, second level, and third level. 

In the resolution of the 3-level image grid data structure, the top level image grid 
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is the lowest, the second level image grid is an intermediate level, and the third level 
image grid has the finest final resolution. 

The top level image grid is divided into the image region including a plurality of 
local cells in proportion to the aspect ratio (MlxNl) of a width Ml and a height Nl. Each 
cell is expressed as a region representative color(RRC) which represents each region, and 
a reliability score(S) which corresponds to the accuracy of the representative color value. 

The second level image grid and the third level image grid are divided into the 
image regions including a plurality of local cells in accordance with the dividing state, 
and each cell has a region representative color(RRC) and a reliability score(S). 

For example, when the maximum width M and the height N of the top level 
image grid are 8(=SxS), the maximum width M2 and the height N2 of the second level 
image grid are I6(=16xl6), and the maximum width M3 and the height N3 of the third 
level image grid are 32(=32x32) of the local cells. 

Here, a certain cell Cell(ij) of the third level image grid is expressed as a region 
representative color and a reliability score C 3 M , S\. 

At this time, the number of divisions of each of the image levels of top level, 
second level and third level is determined based on an aspect ratio of width and height of 
the image for accurately expressing the position. Namely, in the case of the lengthy side, 
the lengthy side is uniformly divided, and the shorter side is divided by the divided unit 
of the lengthy side. 

In another method for generating the grid of the image, to increase processing 
speed and to consider only approximate positional information, each vertical and 
horizontal length of the grid may be identically set. 

The image search method using the multilevel image grid data structure will be 
explained. 
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Different images divided into the multilevel image grids are expressed as a 

representative region color(RRC) for representing the region and a reliability score (C) 

for expressing an accuracy of the representative color, and a pair of representative region 

color and reliability are matched to another one, and a cell similarity is computed in 

accordance with the user content-based query for thereby performing an image search. 

The color similarity between two images which is used as a multilevel image grid 

is computed by comparing the cells included in an image grid of each level and each 

image grid, and the region color(RRC) representing each cell. 

The color similarity between two cells is computed by matching a region 

representative color value between the cell Cell! and Cell Cell, to the color similarities 

Color_Sim(FJ?C_of_Ce!],, PJlC_of_CelI 2 ) which represent the similarity. 

A first weight value (a) is multiplied by a square of the reliabilities S_of_Cell, 

and S_of_Cell 3 for the region representative color RRC of two cells so as to obtain a 

color reliability W which is then multiplied by the color similarities Color_Sim. 

Thereafter, the resultant value is summed by the result obtained by multiplying the 

similarity I with respect to a reliability between two cells and the second weight (p). The 

summed value is then divided by a normalized value. 

Thus, the color similarities Sim(Cell 1? Cell 2 ) of the cells Cell, and Cell 2 included 

in the image can be obtained according to the following equation 1 . 
[Equation 1] 

Sim (Cell &IU ) = axW x Color ~ Sim (RRC j>f Sell, , RRCj>fjCell 2 ) + fixl 
P 2 a + fi 

In the equation 1, the color reliability W which represents how similar the region 
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representative color RRC value between the two cells Cell, and Cell 2 is can be indicated 
according to = (S_of_Cell x x S_pf_Cell 2 ) . The similarity (I) of the reliability which 
represents how reliable the reliability between two cells is can be indicated according to 
/ = 1 -(S _of __Cell x - S _of _Cell 2 ) 2 . 

The weight values (a) and (P) are multiplied, respectively, by the color reliability 
W and the similarity I of the reliability. A normalized value is obtained by multiplying 
the two weight values (a+P) to each other. 

Thus, the color similarity can be searched according to difference of similarities 
for representative color values of two color points having a spatial color feature by 
matching each eel] included in two different image grids. 

Conversely, the similaxily of the color feature between two different multilevel 
image grids can be obtained by matching the same levels to different levels of the 
multilevel image and searching the similarity for the color feature of the images. 

Fig. 2 illustrates an embodiment of the present invention, and shows reliability 
between grids of two images I, and I 2 having a 3-level image. 

For two images I, and I 2 , the reference image 1, has top level G, _top 9 second level 
G,_2 Bd , third level G,_3 rd image grids, and the reference image I 2 has top level G 2 _top, 
second level G 2 _2 Ild , third level G 2 J3 rd image grids. 

The reliability Sim(G i3 G 2 ) between each grid level included in the two images is 
searched by crossing between levels, which can be obtained according to an equation 2 as 
follows. 

[Equation 2] 
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Sim (G P G 2 ) =w , x Sim _ofjhe_Exact G] 




+w 2 x Sim_ofjhe_Exact { 
+w 3 x Sim_ofjhe_Exact { 
+w 4 x Sim_of_the_Inter c 

+w , x Sim of the Inter 

— J - — < 

+w 6 x Sim_of_theJnter ( 
+w 7 x Sim _of Jhe_Inter ( 
+vv 8 x Sim jofjhe_Inter c 
+vv 9 x Sim _of Jhe_Jnter c 





G )J$rd level _and G 2 2 ttd level 



G I 3 rdJevcljmdJS 2 3 /-rf /eve/ 



G \ _2nd_tcveljwdjG 2_3 rrf level 




G I jop Jevel jind J3 2_3 reijevel 



G 2_2nd Jevel jind G 2_2nd level 



Here, wl~w9 denote weight values for representative color similarities. The 
reliability can be searched by dividing a similarity Sim_of_the_Extract between the same 
image grid levels for the two images I, and I 2 , and a similarity Sim_of_the_Inter between 
different image grid levels thereof 

Then, the similarity Sim_of_the_Extract between the same image grid levels 
included in the two different images can be searched by matching the two images to each 
other as shown in Fig. 3. 

That is, similarities of two cells corresponding to the same levels of the two 
different images are summed together, and the similarities of two cells are summed to the 
summed value by shifting in the horizontal and vertical directions by the difference of the 
aspect ratio (width: height). 

At this time, the number of the matches of two grids is computed by adding 1 to 
the absolute value of the difference of the aspect ratio of a certain level of two images. 

For example, for the two images Ij and I 2 , if the width of the image I t is M and its 
height is N, and the width of the image I 2 is O and its height is P, the total number of 
matches between two grids is (|M-0|)+l)x(|N-P|+l). 

The similarity between two cells corresponding to the same grid levels 
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Max(M,N)=Max(0 5 P) is calculated by matching two grids, as shown in Fig. 4, based on 
different shift amount in accordance with the aspect ratio (width: height) of two grids. 

At this time, the similarity Sim_pf_the_Exact based on the match between the 
same levels of two images I, and I 2 is obtained based on the following Equation 3. 

[Equation 3] 

Sim_pfjheJ£xact = Max (Similarity Jietween Jwo Jevelsj*iven j;ellj:orrespondence S(iJ)) 
Vi,0<i<|M-O| 
Vj,0<i<|N-P| 

Similarity_between _two jGr ids _cell_cor respond ence S(i, j ) 

Mi njN- P|-l Mi"n|M-0|-l 

2] ( X Similarity *_of corresponding Jwo jcells (x >y J J) 
Min{N ,P)xMin(M ,0) ~~~~~ ~~ 

When matching the similarity (Sirn_of_the_Exact) between the same levels, the 

Min (\N -P \-\ ) Min (\M -CJ j-i ) 

X ( ^Simj>f_corresjwo_cells) represents a sum of the matching with 

y =0 .v =0 

M-0\ 

respect to the width and height of two corresponding cells. £ X denotes the sum 

j =0 / =0 

for the shift amount. 

The similarity Sim_of_corrresponding_two_cells between two cells is obtained 
from the following Equation 4 by adapting Equations (1)(2)(3)(4) based on the aspect 
ratios (M,N), (0,P). 

[Equation 4] 

Sim(cel! C] (x + 1, y + j ), cell 02 (x, y)\ if(Mm(N 7 P)=P)n (Min(M, O) = 0) 

Sim(cell G] (x + i,y),cell G2 (x,y + j)) y if(Min(N,P) = N)r\{Min(M ,0) = O) 
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Simicell^ix.y + ilcel/^ix + i^v)), if(Min(N,P) = P)n(Min(M 9 0) = M) 

Sim(cell G] {x,y),cell G1 {x + i,y + /')), ifiMiniN.P) = N)r^(Min(M,0) = Af) 

Here, the equation (1) is applied when the aspect ratio (P:0) of the second grid G, 
has the minimum value (min=P:0) with respect to the aspect ratio N:M of the first grid 
G,. Equation (2) is applied when the horizontal length N of the first grid G, is short (min) 
and the vertical length O of the second grid G 2 is short. In addition, Equation (3) is 
applied when the vertical length P of the second grid G 2 is short (rain), and the horizontal 
length M of the first grid G, is short, and Equation (4) is applied when the aspect ratio N 
of the first grid G, is longer than the aspect ratio P of the second grid G,. 

The shift amount (i j) with respect to the length difference (|M-0|,|N-P|) between 
the length of the first grid G, and that of the second grid G 2 is added to the cell coordinate 
(x, y), and each of start point (i,i,x,y) becomes 0. 

The similarity Sim_of_the_Inter between different grid levels (Max(M,N) 
^Max(0,P) is calculated by matching two different image grid levels. The operation is 
performed similarly as the search of the grid level similarity Sim_of_the_Exact. 

In addition, the number of the matches of the image grids between different image 
grid levels is obtained based on (|M-0|+l)x(|N-P|+l) by considering the aspect ratio 
(M,N)(0,P). 

The color region matching operation is performed for searching the region in 
which the representative color values are similar between the multilevel image grids. The 
search is performed based on a method for searching the color similarity from a 
translation position and a relative position between the grid level (Exact scale matching) 
of the same size, and a method for searching the color similarity from a translation 
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position and the relative position between the grid levels (Inter-scale matching) of 
different sizes. 

The color region matching operation between the image grid levels(Exact scale 
matching) of the same size is performed based on a method for searching a color region 
of the same levels from a target image. The position is matched with the relative position 
based on the same image grid level of the target image, and then the similarity of the 
color region is computed, and the position is matched with a translation position at the 
same level of the target image for thereby computing a similarity of the color region. 

The color region matching operation between the different image grid 
levels(Inter-scale matching) is performed based on a method for searching the different 
level color regions among different image grid levels of the target image. 

In the color region matching method of between different image grid levels, the 
similarity of the color region is computed by matching the position with the same 
position among the different image grid levels of the target image, and the similarity of 
the color region is computed by matching the position with the translation position at 
another level of the target image. 

[Effect of the invention] 

As mentioned above, in the present invention, one image grid data structure is 
divided into multilevel grid data structures. Therefore, it is possible to effectively 
response with respect to a subjective query by a user when searching a content-based 
image using the divided multilevel grid structures. In addition, an image search speed is 
fast and accurate under a certain condition. 
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What is claimed is: 



1. A multilevel image data structure in which a spatial color feature of one 
image expressed in a still image is expressed in an image grid having at least more than 
two different hierarchical levels. 

2. The structure of claim 1, wherein the hierarchical image grid of each level 
is characterized by hierarchically dividing resolution of cells in proportion to height and 
width of the corresponding image grid, and each cell is expressed based on a reliability 
on a region representative color and a similarity of the region representative color, 

3. The structure of claim 1, wherein the hierarchical level of each image grid 
is hierarchically divided by uniformly dividing an aspect ratio when an original image 
has a quad tree structure. 

4. The structure of claim 1, wherein in the hierarchical level of each level, 
for non-quad tree structure, one side of the aspect ratio is uniformly divided, and the 
other side thereof is hierarchically divided by the dividing unit of the one side. 

5. An image search method using a multilevel image data structure capable 
of searching images according to a user's content-based query by matching a spatial color 
feature of a reference image divided into different hierarchical image grid levels and a 
color similarity of a target image. 

6. The method of claim 5, wherein the color similarity between two images 



having different hierarchical grid levels is searched according to difference of similarities 
for representative color values of two color points having a spatial color feature by 
matching each cell included in two different image grids. 

7. The method of claim 5, wherein the color similarity between the two 
images having the different hierarchical grids is obtained by matching two image grids, 
performing a multi-cross in accordance with a spatial color feature between the two 
images, and searching color similarities. 



8. The method of claim 5, wherein the color similarity between the two 
images having the different hierarchical ^rids is obtained by matching each region 
representative color value for thereby searching the same regions, 

9. The method of claim 5 or 7, wherein the color similarity between cells 
included in images having different hierarchical levels is obtained by multiplying a 
square of the reliability representing a similarity of the region representative colors 
between two cells which are target images by a weight, adding another weight to the 
similarity of the reliability representing the similarity of reliability between two cells, and 
normalizing the color similarity. 

10. The method of claim 5 or 7, wherein the color similarity between 
multilevel images is obtained based on the total value summed as the values are shifted in 
a horizontal and vertical direction based on the shifting amount by the difference of the 
aspect ratio corresponding to two cells when the same levels of target image grids are 
matched, and a similarity thereof is searched. 
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11. The method of claim 5 or 7, wherein a color similarity between the 
multilevel images is obtained based on a value summed as the values are shifted in a 
horizontal and vertical direction by the difference of the aspect ratio corresponding to two 
cells included in each level in the case that different levels of two images are matched, 
and then a similarity is searched. 

12. The method of claim 5, wherein a cell similarity between image grids 
having a multilevel is used for searching the same position and different position between 
the same levels between the images in the case that the search is performed by matching 
the color region. 

13. The method of claim 5, wherein a color region matching operation 
between two image grids having a multilevel is directed to searching at the same position 
of different levels and at different position when searching the color similarity between 
different levels. 
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